Storage fabric address based data block retrieval

ABSTRACT

Techniques for retrieving data blocks are provided. In one aspect, a storage fabric address of a controller associated with a data block is retrieved by a node. If the node is on the same storage fabric as the retrieved address, the data block may be retrieved over the storage fabric. In another aspect, a directory server maintains mappings of data blocks to storage fabric addresses of controllers associated with the data blocks. A request for the location of the data block includes the storage fabric address of the associated controller.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a divisional of co-pending U.S. patent applicationSer. No. 13/460,115, filed Apr. 30, 2012, the entire contents of whichare hereby incorporated by reference as though fully set forth herein.

BACKGROUND

A distributed file system may distribute the data blocks that make up afile amongst storage devices that are controlled by different dataservers. All of the data servers may be connected via a local areanetwork. A directory server may maintain a list of files and the logicalblock address of each block of data corresponding to a file. Thedirectory server may maintain the local area network address of all dataservers that control storage devices which contain each block of data.In some cases, the blocks of data that make up a file may be replicatedon several different data servers. In such cases, the directory servermay maintain the local area network addresses of all data servers thatcontrol storage devices that contain each block of data.

A client application that wishes to retrieve a file may send a query tothe directory server over the local area network to determine the localarea network address of the data server(s) that contain the blocks ofdata that make up the file. For each block of data, the clientapplication may open a connection to a data server over the local areanetwork using the address retrieved from the directory server. Thecontents of the data block may be retrieved from the storage devicecontrolled by the data server and then sent to the client applicationover the local area network.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an example of a distributed file system.

FIG. 2 depicts another example of a distributed file system.

FIG. 3 depicts an example of a high level flow diagram for retrieving adata block.

FIG. 4 depicts another example of a high level flow diagram forretrieving a data block.

FIG. 5 depicts an example of a high level flow diagram of querying adirectory server.

FIG. 6 depicts another example of a high level flow diagram of queryinga directory server.

FIG. 7 depicts an example of a high level flow diagram of retrieving adata block.

FIG. 8 depicts another example of a high level flow diagram ofretrieving a data block.

DETAILED DESCRIPTION

A distributed file system allows for the data blocks that make up a fileto be spread amongst storage devices controlled by different dataservers. In addition, the data blocks may be replicated, such that thesame data block may exist on storage devices that are controlled bydifferent servers. Such distribution and replication may provide forimproved file handling operations by providing multiple locations fromwhich a block of data may be retrieved. In addition, the replication mayprotect against the unavailability of a data block due to the dataserver that controls the storage device containing the data blockbecoming unavailable.

For example, one currently popular form of a distributed file system isthe Hadoop Distributed File System (HDFS). In one example implementationof the HDFS, each data block of a file is replicated on storage devicescontrolled by three different data servers. Two of the data servers maybe contained in the same rack of data servers, while the third may be ina different rack. In some cases, the racks themselves may begeographically distributed. The data servers may be connected to a localarea network. A directory server, referred to as a name server, may alsobe present on the local area network and keeps track of the local areanetwork addresses for all of the data servers that contain each block ofdata. A client application may be an application that runs on a clientnode. A client node is a server that runs applications. In many cases,the client nodes may be combined with the data servers, such that thedata server both controls associated storage devices and runs clientapplications.

In operation, a client application that wishes to access a file may senda request to the name server over the local area network to determinewhere the data blocks that make up the file are stored. The name servermay then retrieve the local area network addresses of data servers thatcontain the data blocks. The client application may then retrieve a datablock by establishing a connection with the data server over the localarea network and requesting the data block. The data server may thenreturn the data block over the local area network.

Although the system described above is operable, a problem arises in theefficient use of local area network bandwidth. In a typical data centerthat contains may data nodes and client applications running on clientnodes, local area network bandwidth is a limited resource. The bandwidthavailable must be shared by all nodes on the local area network. Thelocal area network may be used for many other actions, such as internode communication, that are unrelated to the distributed file system.Thus, as more local area network bandwidth is used by the distributedfile system, less bandwidth is available for all other users of thelocal area network. The reduction in bandwidth available may increaselatency for not only distributed file system operation but for all otheroperations as well.

The techniques provided herein provide for a mechanism to remove some ofthe distributed file system's traffic from the local area network. Intypical data centers, storage devices, such as hard disk drives, areconnected to data servers over a storage fabric that is isolated fromthe local area network. For example, one such storage fabric is a SerialAttached Small Computer Systems Interface (SAS) fabric. There may beseveral independent storage fabrics within a data center. For example,all servers within a single rack may be connected to the same storagefabric, thus creating a separate storage fabric for each rack.

When a client node wishes to retrieve a block of data, the client may,just as before, send a request to a directory server to find the localarea network address of the data server that controls a storage devicethat contains the block of data. However, in addition to returning thelocal area network address of the data server, the directory server mayalso retrieve the storage fabric address of the controller that controlsthe storage device containing the data block. If the client node and thedata server are both connected to the same storage fabric, the clientapplication may request the data block directly from the controllerusing the storage fabric. The controller may retrieve the data block andreturn it to the client over the storage fabric. Thus, both thetransmission of the request and the actual data block occurs on thestorage fabric, which reduces the amount of bandwidth used on the localarea network. If the client node and the storage device are not on thesame storage fabric, the data block may be retrieved as above, over thelocal area network.

FIG. 1 depicts an example of a distributed file system. The system mayinclude client/data nodes 110, 120, 130. Although only three nodes areshown, this is for simplicity of description. There may be any number ofnodes. The system may also include directory server 140. For purposes ofdescription, only a single directory server is shown. However, thetechniques presented herein are not limited to implementations with asingle directory server. Connecting the client/data nodes and thedirectory server may be a local area network 150. The local area networkmay permit the client/data nodes and the directory server to communicatewith each other.

The system may also include storage devices 170-1 . . . n. Storagedevices may be of any type that is capable of storing blocks of data.For example, some typical storage devices may include hard disk drives,solid state disk drives, optical drives, tape drives, volatile andnon-volatile random access memory, or any other type of storage devicecapable of block level storage. The storage devices may be connected tothe client/data nodes over storage fabrics 160-1,2. The storage fabricsmay allow for communications between any devices that are connected tothe storage fabrics. As shown, all storage devices are not necessarilyconnected to all storage fabrics. For example, storage device 170-1 isconnected to storage fabric 160-1, which in turn is connected toclient/data nodes 110, 120. Thus, these devices may communicate overstorage fabric 160-1. However client/data node 130 is not connected tostorage fabric 160-1. As such, client/data node 130 does not directlycommunicate over storage fabric 160-1.

In operation, a client application running on one of the client/datanodes may wish to access a file. The client may access the directoryserver to determine what blocks of data make up the file. The directoryserver may return to the client a storage fabric address of theclient/data node that is associated with a storage device that containsa block of data that is part of the file. The directory server may alsoinclude the local area network address of the client/data node that isassociated with the storage device.

The client application may then determine if the client/data node thatis running the client application is connected to the storage fabricthat includes the storage fabric address returned by the directoryserver. If so, the client application may retrieve the block of dataover the storage fabric. If not, the client application may open aconnection to the client/data node over the local area network andretrieve the block of data over the local area network.

FIG. 2 depicts another example of a distributed file system. Theelements shown in FIG. 2 are generally the same as those in FIG. 1, witha little bit more detail depicted. Just as above, the system containsclient/data nodes 210, 220, and 230. The system also contains directoryserver 240. The system also contains a local area network, which asshown is an Ethernet network 250. The system also contains storagefabrics, which as depicted are SAS storage fabrics 260-1,2. Connected tothe SAS fabrics are storage devices 270-1 . . . n.

Client/data node 220 has been shown in greater detail than the otherclient/data nodes. However, it should be understood that the structureof the client/data nodes is generally the same. Client/data node 220 maycontain a processor 222. The processor may be of any type that issuitable for use in a computer. Coupled to the processor 222 may be anon-transitory processor readable medium 224 which contains thereon aset of instructions which if executed by the processor cause theprocessor to perform the techniques described herein. For example, themedium may include instructions 225 for requesting and receivingaddresses for data blocks. The medium may also include instructions 226for retrieving data blocks. Operation of the processor executing theseinstructions will be described in further detail below.

Node 220 may also include a network interface 227. The network interfacemay allow the node to communicate with the local area network. Forexample, in the case of the Ethernet network 250 shown in FIG. 2, thenetwork interface may be an Ethernet card. Communication with thenetwork interface may be through a network address. As shown, in thecase of an Ethernet network, the network address may be an IP address.Node 220 may also include a controller 228. The controller may include astorage fabric address which allows for communication with thecontroller over the storage fabric, which as shown is SAS fabric 260-1.

Connected to SAS fabrics 260-1,2 may be a plurality of storage devices270-1 . . . n. As described above, there are many different types ofstorage devices. In some cases, several storage devices may be containedin a single enclosure and the enclosure is attached to the SAS fabric.Regardless of the particular form factor for the storage devices,communication with the storage devices occurs over the SAS fabric.

Storage devices are typically associated with a single controller thatcontrols reading and writing data to the storage device. In order toenforce the single controller for a storage device, some SAS fabricsimplement SAS zoning. When using SAS zoning, a controller is assigned toa zone group. Storage devices may also be assigned to a zone group. Acontroller communicates with storage devices that are in the same zonegroup as the controller. By only assigning a single controller to a zonegroup, it can be ensured that only that controller may read and/or writedata directly to the storage device.

A controller and storage devices that are in the same zone group may besaid to be associated with each other. Thus, when referring to a datablock stored on a storage device, it may also be said that the datablock is associated with the controller associated with the storagedevice. For purposes of description, three zone groups 271, 272, and 273are shown. Storage devices and controllers within the same zone groupmay communicate directly. Thus, as shown, controller 228 may communicatewith storage device 270-2 directly. However, controller 228 may notcommunicate directly with storage devices 270-1 and 270-n. It should beunderstood that there may be any number of zone groups.

Controllers in different client/data nodes may communicate with eachother if they are attached to the same SAS fabric. For example, thecontrollers in nodes 210 and 220 are shown as being connected to SASfabric 260-1, and thus may communicate with each other. However, thecontroller of node 230 is connected to SAS fabric 260-2. Thus, thecontroller in node 230 may not communicate with the controllersconnected to SAS fabric 260-1.

Directory server 240 may also include a processor 241. The processor maybe any type of processor suitable for use in a computer. Coupled to theprocessor may be a network interface 242. The network interface mayallow for the directory server to communicate with the nodes over thenetwork 250. In addition, the processor may be coupled to anon-transitory processor readable medium 243 which contains thereon aset of instructions which if executed by the processor cause the processto perform the techniques described herein. The medium 243 may includeinstructions 244 for receiving and responding to requests for addresses.Operation of the processor will be described in further detail below.

The directory server may also include addresses 245. As has beenexplained above, in a distributed file system, a directory server maymaintain mappings of files to the data blocks that make up the file.Each data block is associated with a local area network address of anode that contains a controller in the same zone group as the storagedevice containing the block of data. The functionality of maintainingthe data block to local area network address is part of typicaldistributed file systems. The techniques described herein modify thedata block mapping to include an additional piece of data. In additionto the local area network address of a node, the directory server mayalso maintain the SAS fabric address of the controller associated withthe node.

For purposes of description, an example of the address information 246for a file is shown. File X may comprise three data blocks, labeled A,B, and C. For each of these data blocks, the local area network address,in this case an IP address, is stored. In addition, for each data blockthe storage fabric address of the controller, in this case the SASfabric address, is also stored. In an example implementation, a blockreferenced by the directory server may not have a one to one mappingwith a block on the storage device. A block as referenced by thedirectory server may actually map to several logical block addresses(LBA) on the storage device. The directory server may store the startLBA, the number of LBAs, and the logical unit number (LUN) of thestorage device (not shown). Using this information, all of the logicalblocks on a storage device that make up a block of data may beretrieved. It should be noted that for purposes of ease of description,only a single network address and storage fabric address is shown foreach block of data. However, as described above, the same block of datamay be stored on multiple storage devices and those blocks of data maybe made up of more than one logical block address on the storage device.In such cases, the directory server will store addresses for all nodesand controller through which the data block is accessible as well as theinformation needed to retrieve all of the logical blocks that make upthe block of data, including the LUN. The client/data node may thenselect how the data block is retrieved, as will be described below.

In operation, a client/data node may be running an application program.The application program may determine that it needs to retrieve aportion of a file. The application program may send a query to thedirectory server to retrieve the addresses of the portion of the filethat should be retrieved. For example, an application program running onnode 220 may determine that it needs to read data from file X. Theapplication program may query the directory server to retrieve thelocations of the data blocks that make up file X. In this case, file Xis made up of blocks A, B, and C, which are stored in storage devicesaccessible by using the listed IP and SAS addresses.

In this example, assume that the application program wishes to retrievethe first block of the file. As shown, block A of the file is accessiblevia IP address 0.0.0.2 and SAS address 2. As node 220 is both runningthe application program and contains the controller associated with SASaddress 2, the node may retrieve the data block directly using thecontroller.

A more interesting case is if the application program decides toretrieve data block B. As shown, data block B is associated with acontroller whose associated IP address is 0.0.0.1 and SAS address is 1.These addresses may be received by the application program. The node 220may then determine that it is connected to the same SAS fabric 260-1 asthe controller which is associated with block B, based on the SAS fabricaddress. In other words, node 220 may examine the SAS fabric address,and based on the address is able to determine if the address is on theSAS fabric connected to node 220. In this example, the controller is onthe same SAS fabric. As such, the controller on node 220 may send arequest for the data block to the controller (not shown) on node 210which is associated with block B. The request may be sent directly overthe SAS fabric 260-1. The controller on node 210 may then return thedata block B to node 220, again using the SAS fabric 260-1. As such, nobandwidth on the Ethernet network is used for requesting andtransmitting the block of data.

The application program may then wish to retrieve block C of the file.As shown, block C is associated with IP address 0.0.0.3 and SAS address3. In this case, the controller (not shown) of node 230 which isassociated with block C is not on the same SAS fabric as node 220. Insuch cases, the application program will default to normal distributedfile system techniques. In other words, node 220 will establish aconnection with node 230 over the Ethernet network. Node 220 may requestblock C over the Ethernet network from node 230. Node 230 may retrieveblock C using its controller. The node may then send block C back tonode 220 over the Ethernet network. As such, the block of data is stillretrieved, albeit without the benefits of offloading traffic from thelocal area network.

Although the description above has been presented in terms of reading afile, it should be understood that the same techniques are applicablewhen writing data to a file. When data is to be written to a file, asimilar process may occur. Although the process may have differentsteps, the concepts described above may be used to minimize the amountof traffic over the local area network. In other words, process steps ina write operation that could be done using the storage fabric make useof the storage fabric. The local area network is used otherwise. Asabove, communication may occur directly to the controller if thecontroller is on the same storage fabric. If not, communication occursover the local area network. For purposes of the remainder of thisdescription, file read requests will be described. However, it should beunderstood that the same techniques may be applied to file writerequests.

FIG. 3 depicts an example of a high level flow diagram for retrieving adata block. In block 310 a client node may retrieve a storage fabricaddress for a controller associated with a storage device storing ablock of data. For example, the client node may retrieve the SAS addressof the controller which controls the storage device that contains theblock of data of interest.

In block 320, the client node may determine if the client node isattached to the same storage fabric as the controller based on thestorage fabric address. For example, the client node may examine thestorage fabric address of the controller and determine if the clientnode's own controller is attached to the same SAS fabric, based on thestorage fabric address.

In block 330, the client node may retrieve the block of data from thecontroller over the storage fabric if the client node and the controllerare on the same storage fabric. In other words, if the client node isattached to the same storage fabric as the controller that controls thestorage device containing the block of data, the client node mayretrieve the block of data over the storage fabric, thus bypassing thelocal area network.

FIG. 4 depicts another example of a high level flow diagram forretrieving a data block. In block 410, the client node may retrievestorage fabric addresses for controllers associated with storage devicesstoring a block of data. As described above, in replicated distributedfile systems, copies of a block of data may be stored on multiplestorage devices. In block 410, the storage fabric addresses of all thecontrollers associated with the storage devices may be retrieved.

In block 420 the client node may retrieve local area network addressesassociated with servers associated with the controllers. As explainedabove, a block of data may be stored on storage devices associated withmultiple servers. The local area network address for these servers maybe retrieved in block 420. In block 430, the client node may determineif the client node is attached to the same storage fabric as thecontrollers based on the storage fabric address.

In block 440, the client node may determine an operating performance ofthe storage fabrics and local area network. For example, the client nodemay determine the latency of the local area network as an indicator ofhow loaded with traffic the local area network is. Similarly, theoperating characteristics of the storage area networks may bedetermined.

In block 450, the client node may select the controller and addressbased on the addresses and the operating performance of the storagefabrics and the local area network. As explained above, in some cases,the client node may be able to access a block of data either over astorage fabric or the local area network. If the client node isconnected to the same storage fabric as the controller of a storagedevice that contains the block of data, the client can retrieve theblock of data using the storage fabric. If not, the client may use thelocal area network. In block 450, the client may select a controllerbased on the returned addresses. In some implementations, if the clientnode is able to retrieve the block of data using a storage fabric, thestorage fabric will be selected.

In block 460, the block of data is retrieved from the selectedcontroller using the selected address. If the storage fabric address wasselected, the client node may send a request to the controller over thestorage fabric and the controller may respond by sending the requestedblock of data to the client. If the local area network address wasselected, the client node may open a local area network connection tothe data server that contains the controller associated with the datablock and the data block may be retrieved over the local area networkconnection.

FIG. 5 depicts an example of a high level flow diagram of querying adirectory server. In block 510 a directory server may receive a requestfrom a client node for the location of a block of data. For example, theclient node may send a request for the location of a file that is madeup of data blocks. The request may include a request for the location ofthe storage devices that contain the data block.

In block 520, the directory server may retrieve a storage fabric addressfor a controller associated with a storage device containing the blockof data. As mentioned with respect to FIG. 2, the directory server of adistributed file system is responsible for maintaining the locations ofthe data blocks that make up a file. In addition to storing the localarea network address of the data servers that control storage devicescontaining a block of data, the directory server may also maintain thestorage fabric address of the controllers. In block 530, the storagefabric address may be sent to the client node. The client node may thenretrieve the block of data as described above.

FIG. 6 depicts another example of a high level flow diagram of queryinga directory server. In block 610, just as above, a directory server mayreceive a request from a client node for the location of a data block.In block 620 a storage fabric address for a controller associated with astorage device containing the data block may be retrieved. In block 630,the storage fabric address may be sent to the client node.

In block 640, a local area network address associated with the serverassociated with the controller may be retrieved. In block 650, the localarea network address may be sent to the client node. As described above,the client node may select one of the storage fabric address and thelocal area network address to use to retrieve the data block. AlthoughFIG. 6 describes separate block for sending the storage fabric addressand the local area network address, it should be understood that theseblocks may be combined into a single block. In other words, thedirectory server may retrieve both the storage fabric address and thelocal area network address associated with a controller and server, andsend both addresses to the client node at the same time. The client nodemay then select which address to use, as described above.

FIG. 7 depicts an example of a high level flow diagram of retrieving adata block. In block 710 a request may be sent to a directory server toretrieve a storage fabric address of a controller associated with a datablock. In block 720, a response may be received from the directoryserver containing the storage fabric address. In block 730 it may bedetermined if the processor that sent the request is associated with astorage fabric including the storage fabric address. In other word, itmay be determined if the processor is connected to the same storagefabric as the controller of the storage device that contains the datablock. In block 740, a request for the data block may be sent to thecontroller based on the determination. If the processor is determined inblock 730 to be on the same storage fabric as the controller, the datablock may be retrieved directly over the storage fabric, thus bypassingthe local area network.

FIG. 8 depicts another example of a high level flow diagram ofretrieving a data block. In block 810 a request may be sent to adirectory server to retrieve a storage fabric address and a local areanetwork address of a controller and a server associated with a datablock. In block 820, a response may be received from the directoryserver containing the storage fabric address and the local area networkaddress.

In block 830, it may be determined if the processor is associated with astorage fabric including the storage fabric address, wherein thedetermination further includes selecting the storage fabric or the localarea network based on the operating performance of the storage fabricand the local area network. As explained above, in some exampleimplementations, the operating performance of the storage fabric isalways considered better than the local area network, and will beselected whenever possible. In block 840, the result of thedetermination is examined. If the determination results in the storagefabric being selected, the process moves to block 850. Otherwise, theprocess moves to block 870.

In block 850, a request for the data block may be sent to the controllerover the storage fabric based on the determination in block 840. Inblock 860, the data block may be received over the storage fabric. Inblock 870, a request for the data block may be sent to the server overthe local area network based on the determination in block 840. In block880, the data block may be received over the local area network.

We claim:
 1. A method comprising: retrieving, with a client node, astorage fabric address for a controller associated with a storage devicestoring a block of data; retrieving, with the client node, a local areanetwork address associated with the controller; determining, by theclient node, whether the client node is attached to the same storagefabric as the controller based on the storage fabric address;retrieving, with the client node, the block of data from the controllerover the storage fabric in response to determining that the client nodeis attached to the same storage fabric as the controller; andretrieving, with the client node, the block of data over a local areanetwork in response to determining that the client node is not attachedto the same storage fabric as the controller.
 2. The method of claim 1,wherein the storage fabric is a Serial Attached Small Computer SystemsInterface (SAS) fabric.
 3. The method of claim 1, wherein the local areanetwork is an Ethernet network.
 4. The method of claim 1, whereinretrieving the block of data over the local area network is furtherbased on an operating performance of the storage fabric and the localarea network.
 5. The method of claim 1, further comprising: retrievingstorage fabric addresses and local area network addresses for aplurality of controllers associated with the data block, the storagefabric addresses associated with a plurality of storage fabrics;selecting a given controller from among the plurality of controllersbased on the operating performance of the storage fabrics and the localarea network; and retrieving the block of data from the selectedcontroller.
 6. A non-transitory computer readable medium containingthereon processor readable instructions which upon execution cause aclient node comprising a processor to: retrieve a storage fabric addressfor a controller associated with a storage device storing a block ofdata; retrieve a local area network address associated with thecontroller; determine whether the client node is attached to the samestorage fabric as the controller based on the storage fabric address;retrieve, with the client node, the block of data from the controllerover the storage fabric in response to determining that the client nodeis attached to the same storage fabric as the controller; and retrieve,with the client node, the block of data over a local area network inresponse to determining that the client node is not attached to the samestorage fabric as the controller.
 7. The non-transitory medium of claim6, comprising further processor readable instructions which uponexecution cause the client node to: send a request to a directory serverto retrieve a storage fabric address of a given controller associatedwith a data block; receive a response from the directory servercontaining the storage fabric address of the given controller; determinewhether the client node is associated with a storage fabric includingthe storage fabric address of the given controller; and send a requestfor the data block to the given controller over the storage fabric inresponse to determining that the client node is associated with thestorage fabric including the storage fabric address of the givencontroller.
 8. The non-transitory medium of claim 7, comprising furtherprocessor readable instructions which upon execution cause the clientnode to: receive the response from the directory server containing alocal area network address associated with a server including the givencontroller; and send the request for the data block to the server overthe local area network in response to determining that the client nodeis not associated with the storage fabric including the storage fabricaddress of the given controller.
 9. The non-transitory medium of claim8, comprising further processor readable instructions which uponexecution cause the client node to receive the data block over the localarea network in response to the request for the data block being sentover the local area network.
 10. The non-transitory medium of claim 7,comprising further processor readable instructions which upon executioncause the client node to receive the data block over the storage fabricin response to the request for the data block being sent over thestorage fabric.
 11. The non-transitory medium of claim 6, comprisingfurther processor readable instructions which if executed cause theclient node to select the storage fabric or the local area network basedon the operating performance of the storage fabric and the local areanetwork.
 12. A device comprising: a processor; and a non-transitorycomputer readable storage medium storing instructions executable on theprocessor to: retrieve a storage fabric address for a controllerassociated with a storage device storing a block of data; retrieve alocal area network address associated with the controller; determinewhether the client node is attached to the same storage fabric as thecontroller based on the storage fabric address; retrieve the block ofdata from the controller over the storage fabric in response todetermining that the client node is attached to the same storage fabricas the controller; and retrieve the block of data over a local areanetwork in response to determining that the client node is not attachedto the same storage fabric as the controller.
 13. The device of claim12, wherein the storage fabric is a Serial Attached Small ComputerSystems Interface (SAS) fabric.
 14. The device of claim 12, wherein thelocal area network is an Ethernet network.
 15. The device of claim 12,wherein the retrieving of the block of data over the local area networkis further based on an operating performance of the storage fabric andthe local area network.